A biologically plausible model of human planning based on neural networks and Dyna-PI models

Understanding the neural structures and physiological mechanisms underlying human planning is a difficult challenge. In fact it is the product of a sophisticated network of different brain components that interact in complex ways. However, some data produced by brain imaging, neuroanatomical and neurophysiological research, are now beginning to make it possible to draw a first approximate picture of this network. This paper proposes such a picture in the form of a neural-network computational model inspired by the Dyna-PI models (Sutton, 1990). The model is based on the actor-critic reinforcement learning model, that has been shown to be a good representation of the anatomy and functioning of the basal ganglia. It is also based on a “predictor”, a network capable of predicting the sensorial consequences of actions, that may correspond to the lateral cerebellum-prefrontal and rostral premotor cortex pathways. All these neural structures have been shown to be involved in human planning by functional brain-imaging research. The model has been tested with an animat engaged with a landmark navigation task. In accordance with the brain imaging data, the simulations show that with repeated practice performing the task, the complex planning processes, and the activity of the neural structures underlying them, fade away and leave the routine control of action to lower-level reactive components. The simulations also show the biological advantages offered by planning and some interesting properties of the processing of “mental images”, based on neural networks, during planning. On the machine learning side, the model presented extends the Dyna-PI models with two important novelties: a “matcher” for the self-generation of a reward signal in correspondence to any possible goal, and an algorithm that focuses the exploration of the model of the world around important states and allows the animat to decide when planning and when acting on the basis of a measure of its “confidence”. The paper also offers a wide collection of references on the addressed issues.